Development of a situational judgment test to supplement current US air force measures of officership.

Aptitude requirements for US Air Force officer commissioning include completion of a college degree and minimum scores on the Air Force Officer Qualifying Test (AFOQT) Verbal and Quantitative composites. Although the AFOQT has demonstrated predictive validity for officer training, the Air Force has striven to improve predictive validity and diversity. To this end, a Situational judgment Test (SJT) was added to the AFOQT in 2015. SJT development was consistent with recommendations to broaden the competencies assessed by the AFOQT with the goal of providing incremental validity, while reducing adverse impact for historically underrepresented groups. To ensure content validity and realism, SJT development was based on competencies identified in a large-scale analysis of officership and input from junior officers in scenario and response generation and scoring. Psychometric evaluations have affirmed its potential benefits for inclusion on the AFOQT. An initial study showed the SJT to be perceived as highly face valid regardless of whether it was presented as a paper-and-pencil test (with narrative or scripted scenarios) or in a video-based format. Preliminary studies demonstrated criterion-related validity within small USAF samples, and a larger Army cadet sample. Additionally, operational administration of the SJT since 2015 has demonstrated its potential for improving diversity (i.e., reduced adverse impact relative to the AFOQT Verbal and Quantitative composites). Predictive validation studies with larger Air Force officer accession samples are ongoing to assess the incremental validity of the SJT beyond current AFOQT composites for predicting important outcomes across accession sources.

Validation and refinement of the "Airman Comprehensive Assessment": Evaluating competency proficiencies of enlisted members.

Beyond proficiency on occupationally specific tasks, the U.S. Air Force expects members to develop proficiency on institutionally valued "soft skill" competencies (e.g., Teamwork, Communication, and Initiative) throughout their careers. As such, all E1-E6 members are annually evaluated using Behaviorally Anchored Rating Scales (BARS) designed to measure such competencies. Despite mandated use, these Airman Comprehensive Assessment (ACA) scales previously have not been empirically evaluated. To address this gap, we surveyed Air Force supervisors, using a criterion-related sampling methodology to validate the behavioral anchors for each scale. Supervisors identified two subordinates of the same rank/career field who they viewed as having (a) high potential for future success in an Air Force career or, alternately, (b) lower potential for future career success and rated each subordinate on the individual behaviors that comprise the 12 scales. ACA items were intermixed with scale items previously identified as distinguishing top performers in civilian organizations. Results demonstrate scale reliability and generally validate the ACA competency scales as stronger differentiators of supervisor-rated career potential than competency scales developed for civilian organizations. We provide recommendations for re-calibration of scale anchors based on the relative percentage of high vs. low potential members that demonstrate each behavior, and suggest changes to improve correspondence between measured competency proficiency and supervisor-rated career potential.

Tailored Adaptive Personality Assessment System (TAPAS) as an indicator for counterproductive work behavior: Comparing validity in applicant, honest, and directed faking conditions.

The predictive validity of the Tailored Adaptive Personality Assessment System (TAPAS), the U.S. Army's first computer-adaptive personality test incorporating multidimensional pairwise preference items, has been demonstrated for training performance in both the Army and Air Force. While the unique TAPAS format has been described as more resistant to applicant faking than traditional self-report personality measures, evidence regarding the magnitude of applicant score distortion on TAPAS, and how such distortion (if present) may affect reliability and validity, has been limited. To address this gap, the present study compared operational TAPAS scores of Air Force enlisted recruits (administered pre-accession to applicants) to their post-accession retest scores under honest and directed faking ("fake good") conditions (based on re-administration of TAPAS during Basic Military Training). Data are presented on the relationship of applicant pre-accession scores to their retest scores under honest conditions (a form of test-retest reliability) and the magnitude of mean score differences in applicant, honest, and directed faking conditions is documented. Further, the validity of the TAPAS as an indicator for counterproductive work behaviors (CWB) was evaluated. Results indicate that TAPAS scores are relatively stable over time and the TAPAS methodology appears to reduce score distortion. In addition, the results suggest that the validities of the TAPAS scores as CWB correlates are comparable across honest and directed faking testing conditions and generally in line with those found for traditional Likert-type self-report Big Five measures.